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USE OF GENERATIVE-ADVERSARIAL NETWORKS 
WHEN CREATING CONTENT 


Abstract. The application of generative-adversarial networks in the creation of content is studied. Monitoring of 
training, analysis of architectures, determination of internal processes at the level of layers, research of properties of 
latent space, and interaction with it are carried out. Variants of using the specified networks in image generation are 
considered. Special attention is paid to practical implementation aspects, including selecting optimal parameters and 
data processing. The difference between a classifier and a discriminator is formulated. The principles of generative- 
adversarial networks and their influence on the efficiency and quality of generated images are studied. The advantages 
and limitations of using GANs in content creation are considered. 

Keywords: generative adversarial networks, discriminator, generator, image generation, TensorFlow 2. 


Introduction 

A popular tool in the modern 
technology industry for content creation is the 
use of generative-adversarial networks (GAN) 
[1]. Algorithms based on machine learning 
principles are capable of generating high- 
quality visual, text and audio content that is 
difficult to distinguish from human-made 
content. 


Analysis of recent research and 
publications 

GANs are used in various fields, from 
art and entertainment to marketing and 
scientific research, opening new horizons for 
creativity. Some of the models can be called 
“creative”. That is why one of the most 
promising areas for generative adversarial 
networks is art and fashion. Well-trained 
GANs can be used to create paintings, songs, 
clothes, and even poems [2]. 

Generative-adversarial networks can be 
used to improve the clarity of images based 
on statistical distributions: they are able to 
predict missing fragments, and generate the 
corresponding pixel values, which will 
improve the quality of photos taken by 
telescopes or microscopes where missing, 
lost, damaged fragments are present [3-5]. 
GANs can be used to predict computational 
bottlenecks in scientific research projects as 


32 


well as in industrial applications. Obviously, 
this is far from a complete list of applications 
of these networks. Despite the interest and 
progress in generative image modeling, the 
successful creation of various high-resolution 
samples from complex datasets is far from 
complete [6, 7]. 


The goal and task of the research: to 
investigate the generative-adversarial 
network, and its internal behavior during 
generation. 

The task of the work is to generate 
content using GAN. It is necessary to find 
architectures and recommendations that will 
improve the quality of the received data. 

Investigate the processes of learning 
models and generating images, and consider 
their internal behavior. Need to: find and test 
different ways to monitor GAN training; find 
out what latent/hidden space is, what its 
properties are, and how to interact with it; 
determine the processes that take place inside 
the obtained neural networks at the level of 
layers; establish and describe the features of 
the mathematical basis of the discriminator; 
formulate the difference between a classifier 
and a discriminator; collect experimental data 
and conduct their analysis. 
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Research materials and methods There are two options for solving the 

Any learning of neural networks, problem: using your PC or an online service, 
including deep learning, is not a simple task for example, Google Colab [9]. The latter 
for a personal computer (PC). Especially if approach does not require additional settings, 
you do not configure the use of GPU has free access to graphics processors, and 
(Graphics Processing Unit) for this process. allows you to distribute documents to other 

The work uses the Python programming people. Proposed technical characteristics of 
language. The TensorFlow framework [7, 8] the free version: T4 GPU, 16 GB of GDDR6 
can work in both modes: if there are no memory at 10 GHz; 12 GB of RAM; CPU, 2 
settings for the use of a graphics card, then cores. 
the calculations are performed on_ the 
processor. Presenting main material 

For small networks, the CPU (Central As is well known, Generative 
Processing Unit) will suffice, but due to its Adversarial Networks are a type of deep 
operational characteristics, it may take learning model that consists of two competing 
significantly longer. It is still recommended to neural networks: a generator (G) and a 
make an effort to install the appropriate discriminator (D). Random noise is applied to 
programs and drivers required for the the input of the generator, and either the 
TensorFlow framework [7, 8]. It is worth output G or real data is supplied to the 
noting that since version 2.11 TensorFlow is discriminator [10]. The output D is the label 
not supported by the Windows operating (probability value) of the class - the data is 
system, so further use requires additional real or artificial. The utility function is 
settings or switching to another system. maximized: 

U(D,G) = Ex~Px(x)[logD(x)] + EZ~Pz(z)[log(1-G(z))]. (1) 

Various training data are used for By definition, the input to GAN 
experimental research, including the MNIST networks is a vector or other form of array of 
(Modified National Institute of Standards and random data, on the basis of which images are 
Technology) dataset [11]. generated (Fig. 1). 

(2, 3) 
Generator 
“Signs” 
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Hidden space swe 
5p Discriminator 


Fig. 1. Scheme of the process of recording features in the hidden space 


GAN research will begin with the equal to two, which corresponds to two- 
architecture selection process. In the work, dimensional coordinates. A size of one 
experiments were conducted with various hundred or more is usually specified, which 
models, let's focus on some of them. The may simplify the training of the model, but 
peculiarity of the architecture of the network complicate further study due to the 
generator (Fig. 2) is the size of the input dimensionality problem. 


noise: the dimensionality of the input data is 
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Fig. 2. Architecture of the 2D generator 


The output value of the generator is 
equal to the data size of the MNIST dataset — 
28 x 28 x 1, which corresponds to the input 
value of the discriminator (Fig. 3). 

As for architectures of generators for 
3D, 4D noises, their changes concern only the 
input layer to the values “?x3” and “?x4”, 
respectively. The discriminators, in turn, do 
not change. 

Training any model can be done using 
the same approach. Only the input-output 
data, the structure of the models, the method 
of storing intermediate results, or their very 
type are changed. 

The latter allows you to have a single 
template, a set of your libraries or packages, 
with which you can reduce code duplication. 
Therefore, 2D GAN training approaches are 
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described below, which will work for other 
models as well, but with their modifications. 
Here are the main stages. 

The work process begins with the 
preparation of input data: normalization or 
standardization. The formation of the 
generator and discriminator, the function of 
calculating their errors, takes place. The 
module is responsible for the direct training of 
models, updating their weights with the help 
of optimizers. 

An important point is to save the state 
of the model to be able to pause the training 
and continue it later. The last template 
component will be the training launch block. 

Individual changes and settings are 
made for each type of model [12]. 
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Fig. 3. Architecture of 2D, 3D, 4D discriminator 


Results of experiments 
The following results were obtained 
during direct GAN training: 

— training was conducted for 15 hours, 
where one epoch accounted for approximately 
10 seconds; 

— a total of 500 iterations of 10 epochs 
were passed, from each iteration 16 photo 
numbers with the same set of noise from 16 
vectors were saved (images from the first and 
last iteration are shown in Fig. 4a and 4b, 
respectively), maps of areas obtained by 
classification of 10,000 generated images by a 
combination of a hundred points in the range 
from -1 to | along the abscissa and ordinate 
axes (therefore, in a combination of 10,000 
noise values). The initial and final state of the 
class map is shown in Figure 5a and Figure 
5b, respectively; 

— the accuracy of the strong independent 
classifier (IC) at the thousandth epoch was 
0.49% on the training set and 0.55% on the 
test set, at the end of training (five thousandth 
epoch) the values were 0.73% and 1.54%, 
respectively. IC is essentially a_ trained 
discriminator, but it is an independent model 
whose task is to distinguish the generated 
images from the real ones with the maximum 
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percentage. The latter distinguishes him from 
a discriminator who does not learn to the end. 
The training schedule of the last strong IC 
(Fig. 6); 

— the accuracy value of a small network 
of an independent classifier (Fig. 7); 

— the history of — generator 
discriminator errors separately (Fig. 8); 

— the result of the generation of the 
trained model (Fig. 9). 


and 
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Fig. 4. Generated images on the first and last iteration 


It is not difficult to notice that the 
quality of the numbers has become better over 
time (Fig. 4). From epoch to epoch, the 
generator improved its output step by step, 
until it reached a stable version, although 
some numbers look rather imprecise, unclear, 
have extra pixels. 
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Fig. 5. Maps of class areas on the first and last iteration 


Analysis of the obtained results makes it 
possible to note an interesting fact regarding 
the map of class regions: the number “1” was 
often found in the center between different 2D 
GAN models. This is what was discovered 
during the restart of training and the process 
of preparing for the final version of the 
network. The explanation may be that this 
figure has the greatest connection with others, 


and its shape easily transforms into most 
others. 

It can be concluded that in the center of 
the map of the hidden space of the model, the 
most common version of the image is most 
often found, which is the carrier of most of 
the characteristics and features inherent in all 
other objects. 


Training and Validation Loss 


20 


—— Taining Loss 
—— Validation Loss 


30 40 50 


Epoch 


Fig. 6. The process of strong learning independent classifier 


After the hundredth epoch, weak ICs 
stopped rapidly increasing in value, as shown 
by the graph in Fig. 7. 

One would assume that the training is 
complete, but this is a misleading impression. 
The ideal variant of the error value in such a 
problem is equal to 50%, so that it can mean 
“suessing” by the classifier: where is one 
class and where is the other. 


The classifier quickly learns to 
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recognize which image is real and which is 
artificial (Fig. 6). 

After the two thousandth epoch, the 
errors gradually move away from each other 
(Fig. 8), which indicates the stabilization of 
learning. The model of a weak independent 
classifier gradually deteriorates the ability to 
separate real images from generated ones, 
therefore, the model has learned (Fig. 7). 
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Fig. 8. History of generator and discriminator errors 


Testing on the validation set with a 
strong classifier showed an error of 0.55% at 
1000 epochs and 1.54% at 5000 epochs, 
which may indicate an improvement in 
generation. The learning stop occurred due to 
a decrease in the intensity of changes in the 
last epochs, which are visible in the sixteen 
image generations from each epoch and in the 
class regions. 

The description of the 3D GAN training 
process now does not require such detail, but 
only requires specific key changes. Thus, the 
convenient view of a rectangular map of class 
areas in 2D space is replaced by a 3D cube, 
which makes it much more difficult to view. 
Such a figure is not easy to visualize; 
therefore, the concept of sections is 
introduced: 16 “sections” are made along all 
axes of the cube (16 points from -1 to 1 with 
the corresponding step) forming _ three 
projections. With such images at each epoch 
or iteration, the process of changing the map 
of class areas can be displayed. 

The results of the experiments make it 
possible to draw the following conclusions: 
training the model took less time compared to 
the 2D version, while the quality of the 
generated images improved. The numbers 
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have become clearer with neat shapes and a 
realistic look. The errors of weak independent 
classifiers have grown in the same way as in 
the previous model. 

As for the errors of strong classifiers, 
the values are quite different. The 2D 
generator was able to achieve a result of 
1.54%, and the 3D - 3.2%. 
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Fig. 9. An example of generating numbers by 
the GAN model using two-dimensional noise 
(five thousand epochs) 
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The dynamics of errors directly of the 
generator and discriminator models does not 
show any problems. 

The increase in the noise dimension had 
a positive effect on the quality of the models, 
while there was a decrease in the time spent 
and the number of epochs (the latter would 
reduce the number of photos if they were 
stored at each epoch). 

We present the results of experiments 
investigating various properties of the 
resulting networks. Without further delay, 
let's start with two-dimensional space. 

A point can be added to the resulting 
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map of the class areas of the two-dimensional 
GAN model, the movement of which allows 
you to explore the hidden space (Fig. 10). 

There are several areas in the program 
interface: directly generated image 
“Generated image”; a sequence of images 
“Activations for each layer (average values)” 
showing intermediate generation results 
starting from the first layer of the neural 
network to the last; map of class areas “Map 
of class areas”; slider for “x” and “y”’, which 
can be used to change the Sone of the 
point. 
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Fig. 10. Interface for exploring 2D hidden space 


The state of the system is updated when the sliders move (Fig. 11): the point with coordinates 
(-0,84; 0,54) occupied a new area on the map corresponding to the number “9”; coordinates (0; 0) 


correspond to the number “1”. 
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Fig. 11. New system status 
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Other areas, taking into account the black and white version (Fig. 12). 
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Fig. 12. Text designation of class areas 


The work implements the so-called “smooth flow” from one point “A” to another point “B” 
(Fig. 13). 
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Fig. 13. “Smooth flow” of images between two points 


The logic of working with three- 
dimensional space is similar to working with 
two-dimensional space. The proposed 
software interface is presented in fig. 14. As 
in the previous approaches, there are regions 
of the generated image in the space, 
sequences of generation by network layers, a 
cube map of class regions, sliders for 
changing the coordinates of a point, the 
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location of which is now tracked not only by 
rotating the cube, but also by using three 
projections on the corresponding planes. 

If you change the coordinates of the 
point, the state of the system will change and 
a new one will be obtained (Fig. 15). The 
point now occupies the corner of the cube 
corresponding to the number eight. 
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Fig. 15. Changing the position of a point in space 


A “smooth flow” between two points is 
also implemented (Fig. 16). 

In the work, a study of CGAN networks 
(Conditional Generative Adversarial 
Networks) is carried out — a type of 
generative-adversarial networks, which during 
generation receive not only noise as input but 
also a condition in the form of a class label of 
the image to be obtained. 

The resulting universal model, which 
can generate more different images per unit of 
noise, theoretically increases the generation 
options several times. 

In a regular GAN, the number of points 
is equal to 10,000, in which all classes are 
distributed, in CGAN — 10,000 for each class 
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separately. 

As disadvantages, one can note: that a 
significant number of images are not unique 
or do not look natural, but this is more a 
matter of the process, learning methods, and 
the structure of the neural network model 
itself. 

As part of the continuation of the 
research topic, the issue of choosing the 
activation function (FA) was considered [13]. 
A series of experiments with the same model 
architecture, but the different most widely 
used FAs, in particular: ReLU, LeakyReLU, 
PReLU, and ELU, were conducted and 
analyzed. 
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Fig. 16. An interface for studying the 3D “flow” of an image between two points 


Testing was performed on 500 epochs 
for each model with the same MNIST dataset 
(Table 1). 

Note that the data in the table are 
obtained using a strong independent classifier. 
Obviously, during training, the results of all 
models grow (the best value is 0.1561), which 
corresponds to a classifier error of 15.61% on 
the test set (Fig. 17). 


Table 1. Results of the experiment with activation 


Fig. 17. Chart of results 


At the end of training, the ReLU 
activation function achieved the best result 
compared to the others. Thus, the choice of 
FA depends on the problem to be solved and 
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functions 
Leaky 
Epoch | ReLU ReLU PReLU | ELU 
100 | 0,0064 | 0,0065 0,0068 | 0,0186 
200 | 0,0198 | 0,0538 0,0473 | 0,0369 
300 | 0,0810 | 0,0502 0,0670 | 0,0967 
400 | 0,0631 | 0,0821 0,0263 | 0,0706 
500 | 0,1561 | 0,1428 0,0789 | 0,0526 
eeeeee -) —— ee 
the designer of the neural network. 
In the work, experiments were 


conducted with different FAs in Landscape 
GAN when generating landscape images. 
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Before that, the model was trained using the 
LeakyReLU activation function, with a 
maximum error of an independent classifier 
of approximately 8%. 

Before the experiment, its architecture 
was taken as a basis and improved, the 
number of convolution filters was increased. 

Table 2 shows some of the experimental 
results obtained. 

With the best result is the ReLU 
activation function, which has 11.45% 
independent classifier error, the result is 
achieved at the four hundred and fiftieth 
epoch. ELU and LeakyReLU are next with 
respective results. 

Each model learned about 20% longer 
than the previous version (100 hours). In 
general, a total of 480 hours were spent on the 
experiment (under the condition of parallel 
training). 


Table 2. Results of the experiment 


Epoch | ReLU ae ReLU | ELU 
10 0,0010 | 0,0036 | 9,86E-04 | 0,0019 
50 0,0288 | 0,0189 0,0057 | 0,0233 
100 | 0,0541 | 0,0626 0,003 0,052 

300 | 0,0689 | 0,0593 0,0063 0,067 
400 | 0,0967 | 0,0725 0,0052 | 0,0587 
450 | 0,1145 | 0,0636 0,0062 0,084 
500 | 0,1086 | 0,0654 0,0035 | 0,0795 
800 | 0,0853 | 0,0458 0,0036 0,099 
1000 | 0,0397 | 0,0718 0,0059 | 0,0825 
Further, the paper investigates the 


behavior of the internal system of models: the 


type of activation between layers in the best 
photo-landscape models with the 
corresponding functions. 

The result is that all activation functions 
try to strongly “cancel” the inputs to the 
neurons. 

Black means the activation value is 


close to zero, and white means strong 
response. 
It turns out that the final image 


generated by the model is created by a large 
number of “dot-sticks”, which are then 
grouped into a photo as the last layer. 

It can be concluded that when changing, 
for example, the value of activations after the 
penultimate layer, you can generate separate 
areas of the photo to improve it (remove 
defects or other artifacts). 

Table 3 contains information about the 
training time of the models, the number of 
epochs, the size of the dataset and images, the 
quality of generation according to the 
assessment of the independent classifier, the 
number of parameters of the models. 

Table 4 summarizes information about 
testing with various activation functions. 

Obviously, the ReLU activation 
function ranks first in both types of generation 
problems. 

For a better understanding of the 
operation of the generator and the hidden 
space, the variational autoencoder (VAE) is 
considered, which, with the help of root mean 
square error and Kullback-Leibler (KL) 
divergence, distributes vertices in the space 
next to each other in order of similarity. 


Table 3. Results of experiments with models regarding the quality of generation, 
which were evaluated by independent classifiers 


G ‘ Numbers (noise) 
erpeor 2D 3D, CGAN 4D 
tlearning, hours 15 10 a) 3 
tavg to 1 epoch, s 10 20 0.125 10 
n epoch, thousands 5 2 60 1 
Dataset, thousands, shape 60, 28 x 28 x 1 
Memory, GB 0,18 
IC, &, % 15 Spe 0,084 8,29 
Total 1101632 | 1114176 | 1114206 | 1 126720 
params|°| Trainable | ! OT 1° | 1088 704 | 1088 734 | 1 101 248 
Non-trainable 25 472 
D Total 212 865 212 865 220 705 212 865 
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Table 4. Results of testing models with different activation functions 


Numbers 
Activation function ReLU LeakyReLU PReLU ELU 
The biggest error, % 15,61 14,28 7,89 9,67 
Study time: one epoch, s—total | 9s-— 1 hours 10 s — 1 hours 10s—1hours | 9s—1 hours 
time, hours 15 min 06 s 22 min 37s 15 min 14s 19 min 32s 
Generator parameters 1 126 720 
Discriminator parameters 212 865 
Landscapes 
The biggest error, % 11,45 9,74 0,92 10,61 
Study time: one epoch, s 410s 410s 430s 420 s 
Generator parameters 10 729 344 
Discriminator parameters 2 563 713 


Assume that the GAN implements this 
behavior without direct instructions. 
Kullback-Leibler divergence [14]: 


Dxt[N(u,0||N(0, 1)=0,5 (1+log(o”)-p2). 


The online service of the Kaggle 
platform was used to search for datasets and 
train networks. 

The architecture of the U-Net network 
was used as an experiment. The latter is done 
to get the “noise” to deep levels. 

Special attention is paid to the selection 
of the dataset. As already noted, the MNIST 
dataset is quite clear and easy to use, but not 
only it was used in the work. 

The Kaggle platform is known to 
contain a sufficient number of available 
datasets for various machine-learning tasks. 
Of course, there are other online services, for 
example, the possibility of using GitHub 
repositories or Alibaba's Tianchi platform for 
this. 

Six thousand images of ships with a size 
of 32x32 pixels were used as a dataset. An 
independent classifier can indicate the quality 
of the generation: the worse the division into 
two classes is, the better the trained GAN 
model. 

An experiment on the convolution 
kernels showed that the best result, with a 
13% IC error, was the value of the 
convolution kernels “5-5”. Therefore, the 
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following U-Net-2 model was trained over 
twenty thousand epochs, which had erratic 
errors, low-quality signal generation, and a 
test with an independent classifier showing a 
low 8%. 

For further experiments, it is suggested 
to consider a linear model and drawings of 
landscapes with a size of 64x64 pixels. To 
improve the quality of training, it is proposed 
to increase the volume of the dataset from five 
to twenty thousand. 

The following model is trained on 
ninety thousand real 64x64 landscape photos. 

The generated color images have 
visually clear images and clearer boundaries 
of objects (Fig. 18). 

As a result, an image with sufficient 
detail and image clarity was obtained. During 
testing with an independent classifier, an error 
of 11% was obtained (Fig. 19), which is quite 
good compared to past experiments. 

Note that the following layers are used 
in the generator model: Dense, 
BatchNormalization, LeakyReLU, Reshape, 
and Conv2DTranspose. 

As already mentioned earlier, as part of 

the work, tests were conducted with various 
functions of activations in the models. 
The best version of the model has ReLU as 
the main FA, and the hyperbolic tangent 
(tanh) on the last layer. Diagram of errors 
(Fig. 20). 
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Fig. 18. Generated Images 


Training and Validation Loss 


Loss 


Epoch 47/600 


—— TFaining Loss 
—— Validation Loss 


71/71 [S=SSSSSSSSSSSsSsSaSssssessssssa| -3s 
40ms/step - loss: 0.0083 - accuracy: 0.9982 - val_loss: 
0.1125 - val_accuracy: 0.9690 - Ir: 5.2429e-05 


Fig. 19. IC test 


The intermediate state of the model at 
the four-hundred-and-fiftieth epoch (Fig. 21) 
and the final state of the model (Fig. 22) are 
given below. 

When visually comparing (Figs. 21 and 
22) the intermediate and final state of the 
network, it can be stated that at the time of the 
four hundred and fiftieth epoch, the images 
have a better display of images than the final 
result. 

The results of experiments conducted 
on training neural networks are given in [7]. 

Note that different network 
architectures, their parameters and_ layers, 
activation functions and datasets are used. 
The resulting data can be used to train 
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networks to generate images larger than 
64x64x3. 


Conclusions 

The work included: the behavior of 
generative-adversarial networks was 
investigated, the hidden space and interaction 
with it were considered. 

The template architecture of the 
generator and discriminator is given. The 
learning processes of 2D, 3D and 4D GAN 
networks are described. 

The behavior of the final versions of the 
models after their training is considered and 
investigated. The features of the generative- 
adversarial network with the CGAN condition 
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are given, and the representation of their of generating images of handwritten numbers 
hidden spaces is considered. and photos of real landscapes. 

Experiments were carried out with 
activation functions when solving the problem 
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Fig. 20. Graph of errors by epoch 


Fig. 21. Intermediate state of the model 
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Fig. 22. Final state of the network 
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